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ABSTRACT 

In large lecture courses, it can be challenging for instructors to address student misconcep¬ 
tions, supplement background knowledge, and identify ways to motivate the various interests 
of all students during the allotted class time. Instructors can harness instructional technology 
such as screencasts, recordings that capture audio narration along with computer screen im¬ 
ages, to supplement the lecture with content that addresses the diversity in student academic 
backgrounds, motivations, and interests, to extend the classroom experience, and reach the 
individualized needs of students. This study documents the strategic use of screencasts in a 
large introductory Materials Science and Engineering (MSE) course, and examines their impact 
on student usage and course performance. To assess the efficacy of screencasts, students were 
surveyed to determine how they used screencasts and whether they perceived these resources 
to be helpful. In addition, we correlated student usage based on website hits with student 
performance (e.g. final grade) to determine statistical significance. Since the course is com¬ 
prised of students from different academic and social backgrounds, we also analyzed usage 
and performance patterns for particular student subgroups. The results indicate that students 
perceive the screencasts to be helpful and tend to use the resources as a study supplement. 
Overall, usage of screencasting in its various forms is positively and significantly correlated 
with course performance as indicated by the final grade. The most substantial gains were found 
for students with the least amount of prior exposure to concepts in the course material. These 
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results indicate a potential for screencasts to address the various academic needs of students 
in a large lecture environment. 

Keywords: Instructional technology, screencasts, conceptual understanding, materials science 


BACKGROUND 


A screencast is a movie that captures the activity on a computer screen with real time audio 
commentary. Screencasting has been successfully implemented as a training tool for task-centered 
activities in a variety of contexts [1-3], Screencasts have been used to record and replay Power¬ 
Point lectures and other training materials, but any application—including animations or drawing 
programs—may be captured along with audio commentary, then posted on a course management 
website and replayed at students’ leisure. Used properly, this approach can be a rich supplementary 
resource. It includes written, audio, and video elements, thus reaching students with many different 
learning preferences. Screencasts may be fast-forwarded, rewound and paused as necessary, and 
are available “on demand” and accessible to students at any time of day (in contrast to office hours, 
live chats, etc.). Our implementation of the screencast creates a flash movie that can be accessed 
through the course management website, but students may also elect to subscribe to the screencast 
via iTunes, allowing students to watch them on any mp3 player. The production of screencasts is 
extremely simple thanks to powerful software packages now available that range in both price and 
capability [4], 


LITERATURE REVIEW 


Instructors are using audio and visual podcasts or screencasts to replace lecture, to supplement 
lecture material, or to assign projects for which students create video podcasts [5], Perhaps the 
most straightforward and widely adopted use of screencasting is in recording lectures as they are 
delivered in the classroom, and posting the lectures online through a course management website. 
Figure la shows a screenshot of a lecture screencast from a course at the University of Michigan. In 
this case, it includes video of the instructor and of the presentation slides. The use of screencasts 
as a replacement of live lectures has been proposed as a strategy to shift class time towards more 
interactive activities [6], or mitigate the high likelihood of problems during live in-class demon¬ 
strations [7], However, adoption of these technologies has been hampered by the perception that 
providing these resources will result in a drop in student attendance [8], despite early analyses 
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Figure 7: (a) Lecture recording and (b) Homework solution screencast. 
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showing little impact in this regard [9]-[12], Typically students self-report using the posted lecture 
material for review, or to make up absences [13]-[16], 

Initial data suggests that screencasting can be a tremendously useful resource for students. 
Several studies show that screencasting is perceived by students as beneficial, [16]-[20] though 
not a substitute for instructor-student interactions [20], One study analyzed course perfor¬ 
mance by comparing students who used screencasts to students who attended live lectures, 
and determined that students benefited from screencasts especially if they had lower class at¬ 
tendance [21], Grabe and Christopherson have done valuable studies looking at screencasting as 
a resource for note-taking, and they have found positive correlations of screencast usage with 
exam performance [15]. 

The use of screencasts that supplement regular lectures, however, is still an active area of re¬ 
search. Lee, Pradhan, and Dalgarno developed screencasts to scaffold learning of object oriented 
programming in an introductory computer science class [22], In that study they found no statis¬ 
tical significance for the effect of screencasts on learning, perhaps because actual usage of the 
screencasts was not monitored. Other studies show evidence of higher retention of material when 
presented in screencast form rather than as live lectures. Dey and coworkers developed 20-minute 
presentations that emphasize a concept in physics in three formats: a live lecture, a screencast with 
voice narration, a screencast with narration and a video of the speaker [23], They found that stu¬ 
dents viewing the screencasts remembered key concepts better than those who attended the live 
presentation. Students reported a preference for having the image of the speaker in the screencast, 
but its absence had no impact on the retention of material. 

These results are promising; however, more data are needed to explore the impact of screencast 
use on student performance, and to examine the impact of these resources on individual student 
learning. With increasing attention to issues of social equity, it is also important to determine if 
gender, race, native language, or socio-economic status are factors in students’ success in utilizing 
new technologies. With these issues in mind, our study takes a form that builds on and extends 
other analyses of web-based lecture resources. 


SCREENCAST DEVELOPMENT AND BEST PRACTICES 


In this course, screencasts were used in different ways: to capture lectures as they occurred, to 
disseminate solutions to assignments (e.g., homework, quizzes and exams), and to clarify topics that 
the students themselves identified as unclear. While each type of screencast served a somewhat 
different purpose, they are all similar in terms of development and production. 
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The College of Engineering at the University of Michigan has specially equipped lecture halls that 
record and publish lectures automatically. All images projected onto the screen are captured, and a 
camera mounted at the back of the room documents the action at the lectern. Audio is recorded via 
a lapel microphone worn by the instructor, and microphones in the classroom capture the comments 
of the class. These elements are automatically produced into a stream file that may be accessed by 
a course management website, an example of which is shown in Figure la. Such a sophisticated in- 
class infrastructure is not strictly necessary for capturing and posting lectures online. Similar results 
may be achieved by using any number of commercial screen capturing software [4], [7], 

Recent advances in personal computing hardware and software have made the production of 
screencasting easier than ever. All that is necessary is a personal computer capable of running any 
of the currently available screencasting software packages. There are several options that range in 
price and capability, and learning to use them is akin to using basic modern movie editing software. 
In some cases, an external microphone may be advisable if the internal microphone results in unac¬ 
ceptable audio quality. 

There are four aspects of developing screencasts: content preparation, recording, editing and 
production, and publishing. As in any good teaching practice, the success of the resource is based 
on identifying the learning goals for students and preparing content based on these ideals. The 
purpose of the screencast will strongly determine its tone, structure, and content. For example, 
questions that come up in class may be best answered with a quick and casual video, whereas an 
extremely detailed and polished video is more appropriate for homework solutions. In either case, it 
is generally better to keep screencasts short (with the exception of capturing regular lectures), on 
the order of 15 minutes or less. Various elements may be incorporated into the screencast to better 
support its purpose, such as written text and pictures, movies and animations, or applications. Once 
the materials for the screencast are ready, the screencast may be recorded, edited, and produced 
using screencasting software. Another factor that must be considered is how the finished product 
will be disseminated. Options include a closed access course management website, a course blog, or 
a more public video viewing site. In each of these cases, there are slightly different issues pertaining 
to the production quality and copyright issues. For instance, a screencast that will be published on 
a public site should be accurate, high quality, and free from any copyrighted material. Screencasts 
on access-controlled sites, on the other hand, do not necessarily have to hew to such high standards 
and make it easier to protect intellectual property. Zhu and Bergom provide additional recommen¬ 
dations for screencasting, especially lecture capturing [24]. 

The majority of this study was focused on the efficacy of supplemental screencasts as student 
learning tools. Figure 1b shows an example of a homework solution screencast, and a full example 
may be viewed at jmmg5.engin.umich.edu/hw04/hw04.html. This approach is more detailed than 
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annotated written solutions, because it is possible to show the student where in the text or lecture 
notes the concept is described, talk through setting up the problem, and demonstrate how to use 
computational tools, such as Mathematica or Excel, to find the solution. The most novel application 
of this approach, however, is in the creation of supplemental screencasts that clarify topics that 
the students themselves identify as unclear. This approach is adapted from the “muddiest point” 
classroom assessment technique [25], where students are directed to an online survey throughout 
the term to self-assess their confidence on a variety of learning objectives for each unit. The instruc¬ 
tor created the “muddiest point” screencasts when at least 30% of students indicated their need. 
The benefit of these “muddiest point” screencasts is that only the students who need or want the 
supplemental material access it. An example of this type of screencast may be viewed at jmmg5. 
engin.umich.edu/dislocation/dislocations.html. 


ANALYSIS OF SCREENCAST USAGE DATA 


Over the course of four semesters, the authors conducted a study to investigate whether screen- 
casts are an effective tool for student learning in a large introductory lecture course. The study was 
performed in MSE 220, Introduction to Materials and Manufacturing, which averages nearly 200 
students with different backgrounds and motivation. The goal of this research is to determine the 
efficacy of screencasts in MSE 220 by determining how students use different kinds of screencasts, 
which students use them, and whether and for whom screencasts affect learning. We examined the 
efficacy of screencasting in detail for two terms (e.g., Fall 2008 and Winter 2009), although data 
collected in Fall 2006, Fall 2007, and Winter 2010 inform some of the analysis. Every effort was 
made to keep the course the same between the various semesters. The same instructor taught the 
course for all terms using the same syllabus, text book, lecture notes, and homework assignments. 
One difference between the semesters is that previous screencasts were made available immediately 
in the later semester, with additional screencasts added in response to student demand. While mid¬ 
term student feedback and end-of-term course evaluations were used to assess the overall course, 
an end-of-term online survey was also conducted to evaluate students’ perceptions of screencasts 
in particular. With the approval of our Institutional Review Board, the survey asked students about 
how they used the screencasts, and whether these resources were helpful in providing a deeper 
understanding of the course concepts. For Fall 2008 and Winter 2009, the response rate for the 
online survey was greater than 65% (e.g., 68% for Fall 2008 and 66% for Winter 2009). 

For statistical analysis, additional student information was collected (e.g., demographic and usage 
data) for each term in order to correlate these factors with course performance. The office of the 
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registrar provided information on race/ethnicity, gender, citizenship, academic level, major, GPA, and 
SAT/ACT scores for each student. The office administering course management servers provided 
usage data for each student enrolled in the class, including information on which screencasts were 
viewed and when they were accessed; however, no information was available for the duration of 
use for each resource. PASW Statistics 18 software (www.spss.com) was used for all quantitative 
analyses. Correlations were run to explore the relationship between usage and performance for all 
students. Basic descriptive analyses were conducted for usage and performance measures, both 
overall and according to the dimensions of gender, race/ethnicity, major, etc. Additionally, correla¬ 
tions, one-way analysis of variance (ANOVA) and post-hoc analyses were used to test for signifi¬ 
cance in usage and performance means across these various dimensions. In the particular case of 
student academic background, one-way analysis of variance, and post-hoc analyses were used to 
test for significance in the relationship between usage and performance for each of the possible 
majors. Where theoretically appropriate, smaller subgroups were combined to increase sample size 
for these means comparisons. For all ANOVA tests, Levene statistics were examined for violation of 
the assumption of equality of variance. For tests that revealed significance, unless otherwise noted, 
no violation of this assumption was found. When violation of the equal variance assumption was 
found, Welch and/or Brown-Forsythe robust F tests are included in footnotes. 


COURSE BACKGROUND AND STUDENT DEMOGRAPHICS 


Materials Science and Engineering (MSE) is concerned with the study and development of mate¬ 
rials used in virtually every physical process and engineering discipline. Due to its interdisciplinary 
nature, MSE 220 is taken as a technical elective by many students in the College of Engineering at 
the University of Michigan. Student academic backgrounds and class levels in MSE 220 are sum¬ 
marized in Table 1. In general, the populations of students in the two terms examined in this study 
look very similar with regard to the distribution of gender, race and ethnicity, and academic perfor¬ 
mance. The majority of students (.70%) during each term were from three departments: Aerospace 
Engineering (AERO), Chemical Engineering (ChE), and Industrial and Operations Engineering (IOE). 
Students majoring in Nuclear Engineering and Radiological Sciences (NERS) and MSE comprise of 
a small fraction of the student population for both terms (less than 9% and 5% respectively). The 
category for “Other Engineers” includes small numbers of undeclared engineering majors, engi¬ 
neering physics, Electrical Engineering, Mechanical Engineering, and Civil Engineering students. In 
both terms at least 40% of the students were juniors. In particular, AERO students were most likely 
to be juniors. ChE students were split between junior and senior level students. IOE students were 


WINTER 2011 


7 






ADVANCES IN ENGINEERING EDUCATION 

The efficacy of screencasts to address the diverse academic needs of students 

in a large lecture course 




Fall 2008 

Winter 2009 


Number Percentage 

Number 

Percentage 

Sex 

Males 

165 

75.3 

130 

73.0 

Females 

54 

24.7 

48 

27.0 

Total 

219 

100.0 

178 

100.0 

Race/Ethnicity* 

Caucasian 

141 

64.4 

96 

53.9 

African American 

6 

2.7 

14 

7.9 

Asian 

39 

17.8 

28 

15.7 

Hispanic 

2 

0.9 

14 

7.9 

Native American 

1 

0.5 

2 

1.1 

None* 

30 

13.7 

- 

- 

Not Indicated 

- 

- 

24 

13.5 

Total 

219 

100.0 

178 

100.0 

Academic Level 

Senior 

59 

26.9 

76 

42.7 

Junior 

101 

46.1 

75 

42.1 

Sophomore 

59 

26.9 

26 

14.6 

Freshmen 

- 

- 

1 

0.6 

Total 

219 

100.0 

178 

100.0 

Major 

Aerospace (AERO) 

60 

27.4 

36 

20.2 

Chemical (ChE) 

49 

22.4 

45 

25.3 

Nuclear Eng. & Radiological Sciences (NERS) 

19 

8.7 

9 

5.1 

Industrial and Operations (IOE) 

53 

24.2 

60 

33.7 

Material Science (MSE) 

9 

4.1 

5 

2.8 

Other Engineering (Mechanical, Civil, etc) 

26 

11.9 

22 

12.4 

Other (Non-Engineering) 

3 

1.4 

1 

0.6 

Total 

219 

100.0 

178 

100.0 

Cumulative GPA 

Average Cumulative GPA 

3.18 

- 

3.09 

- 

4.0-3.6 

48 

21.9 

31 

17.4 

3.5-3.0 

93 

42.5 

73 

41.0 

2.9-2.6 

48 

21.9 

40 

22.5 

2.5-2.0 

24 

11.0 

29 

16.3 

1.9 and below 

6 

2.7 

5 

2.8 

Total 

219 

100.0 

178 

100.0 

Citizenship Status 

U.S. Citizen 

184 

84.0 

156 

87.6 

Perm. Resident 

13 

5.9 

9 

5.1 

Non-Resident 

22 

10.0 

13 

7.3 

Total 

219 

100.0 

178 

100.0 


* Students who identify themselves as “none of the above” 


Table 1: Student background characteristics for Fall 2008 and Winter 2009. 
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most likely to be seniors in Winter 2009, but had approximately equal proportions of juniors and 
seniors in Fall 2008. AERO, ChE, and IOE students typically had similar mean GPAs, ranging from 
3.1 to 3.3 for both terms. 

A consequence of this wide range of majors enrolled in the course is diversity in disciplinary 
background and motivation of the students. Figure 2 lists the core courses required by MSE and 
the three departments with the largest number of students participating in the course. The bolded 
course names are those that contain topics that are similar to those covered in MSE. ChE, which is 
based on chemical and life sciences, has five directly similar courses, with topics such as thermo¬ 
dynamics, kinetics, and mass transfer. AERO, which concerns itself with the design and fabrication 
of air and space vehicles, has four courses that cover topics related to mechanical behavior and 
thermodynamics, although these courses are very specifically targeted at issues in aerospace and 
less so on material properties. IOE pertains primarily to integrated systems and draws expertise from 
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Figure 2: Comparison of the core courses required by the largest constituents taking MSE 
220 to the Materials Science core. Courses listed in bold are those with content relevant to 
Materials Science. 
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mathematics and social sciences. As such, there are no courses in the IOE curriculum that pertain 
directly, or even indirectly, to topics in MSE. Therefore, despite the fact that on average lOEs have 
comparable academic indicators (e.g., GPA, SAT, ACT) as the rest of the class, they start MSE 220 
at a disadvantage for having less experience with the class material in prior coursework. 


RESULTS AND DISCUSSION 


This work focuses on determining which groups of students use screencasts and whether their 
use improves performance in the course. We focus on screencasts based on solutions to assign¬ 
ments (e.g. homework, quizzes, and exams) and “muddiest point” topics in MSE 220 during Fall 2008 
and Winter 2009 semesters. Previous work on these types of screencasts and lecture recordings 
suggests that students typically used these resources as a study supplement and found them to 
be helpful [17]—[19]. Other studies support this finding [21], For instance, 89% of students from Fall 
2008 and Winter 2009 who responded to the survey and viewed the homework solution screencasts 
(N=126/141) or viewed the quiz solution screencasts (N = 96/108) felt that the explanations for each 
topic were helpful. One student from the Winter 2009 class explained, 

Screencasts both showed solutions and problem strategies. The verbal explanation makes it 
more understandable than a simple [homework] solution handout. 

In Fall 2008, one student said, “Quiz solution screencasts allowed me to quickly see any errors I 
made.” While another student mentioned that “the quiz solution screencasts allowed me to clarify 
concepts I didn’t fully understand before the quiz, and helped me master them before the exam.” 

Unlike the homework or quiz solution screencasts that were created for every assignment, the 
“muddiest point” screencasts were developed when at least 30% of students indicated that they 
wanted additional information to clarify their understanding of course concepts. Eighty-six percent of 
the students (on average) who responded to the survey and viewed the muddiest point screencasts 
felt that the explanations given in the muddiest point screencasts were “somewhat helpful,” “very 
helpful,” or “extremely helpful” during these terms. In Winter 2009, one student explained, 

[The "muddiest point" screencasts] were very clear and the step-by-step explanations 
cleared up any confusion / had. For example, on the phase diagram examples it was really 
helpful to see [the instructor] actually trace the path in question and reference the problem 
to help me to be able to use the text to help solve the problems. 


to 
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Furthermore, students (N=216) rated their level of agreement with the statement, “I have a deeper 
understanding [of course concepts] due to [these resources]” at a 3.3/4.0 or higher for both terms 
(1 = Strongly disagree, 2=Disagree somewhat, 3=Agree somewhat, 4=Strongly agree). One Winter 
2009 student explained, 

/ really liked listening to the solution explanation to the homeworks even if I got the problem 
right. Just listening to the way the problem was reasoned out in words helped me to 
remember processes and procedures better. 

These positive perceptions highlight the value of the screencasts from the student point of view; 
however, they also illustrate the need to discover more specifically which students use screencasts 
and how often they use them, as well as, whether screencasts improve their learning (i.e., final 
course grade). 

Overall Usage and Performance 

Although the overall usage of screencasts was somewhat higher in Winter 2009 than Fall 2008, 
both terms show a positive significant correlation between usage and performance. For the purpose 
of this study, usage data was examined in terms of the number of website hits per student, which 
estimates how often students accessed various screencasts. In Fall 2008, 25 screencasts were devel¬ 
oped and made available in Winter 2009. Eight more screencasts were developed in Winter 2009 in 
response to student queries for a total of 33 screencasts in that semester (Table 2). Students were 
most likely to view the homework solutions and the muddiest point screencasts, and were least 
likely to view the quiz and exam solution screencasts (Table 2). 



Fall 2008 

Winter 2009 


Number of 
Screencasts 

Number of Hits 

Number of 
Screencasts 

Number of Hits 

Homework solution 

12 

2604 

13 

2220 

Muddiest Point 

9 

1405 

15 

2280 

Quiz and Exam solution 

4 

355 

5 

430 

Total Screencasts 

25 

4364 

33 

4930 


Table 2: Number of screencasts and website hits for all types of screencasts in Fall 2008 
and Winter 2009. 
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The screencast usage patterns vary between the two terms and are characterized in Figure 3 by 
the level of screencast use which includes six usage groups ranging from non-users (with 0 website 
hits) to very high users (with over 41 website hits). Less than 8% of students did not access any of the 
screencasts for both terms (Fall 2008:16 non-users out of 219 students; Winter 2009:14 non-users 
out of 178 students). The usage levels depicted in Figure 3 show a large difference in the number of 
very low and very high users between the two terms. Fall 2008 is characterized by having 49% of 
students who are low and very low users (N=107). Winter 2009, on the other hand, has 38% of its 
students as high and very high users (N=67). 

It is not entirely clear as to why the usage pattern varies between these terms. It is possible that 
the variation is due to how screencasts were advertised differently to students and whether students 
were aware of the preliminary research that indicated the helpfulness of screencasts [19], In Fall 
2008, students were not informed about the preliminary research because the data was still being 
tabulated; however students from Winter 2009 had the advantage of knowing the performance 
benefits of screencasts. Furthermore, at the beginning of each semester the previously developed 
muddiest point screencasts were immediately available for students to view. In Winter 2009, students 
had access to nine muddiest point screencasts from the beginning of the term, compared to only 



Level of Screencast Use 

Figure 3: Level of screencast use based on the number of website hits for Fall 2008 and 
Winter 2009. Level of screencast use are defined as follows: non-users (N) with 0 website 
hits, very low users (VL) with 1-10 website hits, low users (L) with 11-20 website hits, mid¬ 
level users (M) with 21-30 website hits, high users (H) with 31-40 website hits, and very high 
users (VH) with more than 41 website hits per term. 
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three in Fall 2008. This provides additional opportunity for students to increase their usage of the 
screencasts. For the purpose of this study, we attempted to reduce other inconsistencies between 
the two terms by designing the classes to have the same instructor who presented lecture material 
relatively consistently; comparable homework, quiz, and exam questions; and similar methods for 
identifying “muddiest point” topics. Although the usage patterns vary between the semesters, the 
student populations are demographically similar (see Table 1), so we have chosen to combine the 
two semesters for statistical analysis. 

The analysis above shows how often students used screencasting in its various forms, however it is 
essential to know whether screencast usage actually improved student performance as indicated by 
their final grades in the course. Indeed, we find that students benefited from using screencasts in both 
Fall 2008 and Winter 2009. A positive significant correlation is found between the number of website 
hits and final course grade: r = .245, p = .000. This modest yet statistically significant correlation shows 
that students who use screencasts more tend to receive higher grades in the course. This result is com¬ 
parable to other researchers’ findings [21], When we control for GPA, the magnitude of this correlation is 
somewhat reduced but the statistical significance is retained: r = .178, p = .000. In other words, assuming 
that GPA is an indicator for expected performance in this course, higher screencast usage still produces 
a higher final course grade when comparing students of similar past academic accomplishment. 

With these general relationships in mind, we look at usage and performance for particular groups 
within the larger student population of MSE 220. Specifically, the dimensions of gender, citizenship, 
race/ethnicity, academic level, and academic major are investigated, but only academic major is 
found to be a significant indicator for usage and performance benefits. 1 

Usage and Performance by Gender, Citizenship, Race & Ethnicity, and Academic Level 

In all, when we examine differences in means for screencast usage (i.e. website hits) and course 
performance (i.e. final course grade) across the dimensions of gender, citizenship, race/ethnic¬ 
ity, and academic level, the results are mixed, stimulating interesting pedagogical questions. For 
each dimension, we explore whether certain subgroups benefit more or less from screencast use. 
As well, we explore whether usage is in fact mediating the observed relationships between being 
in particular subgroups and course performance. Table 3 depicts the mean number of website 
hits, mean final course grade, and number of students for each of these student populations, and 
Figure 4 shows the distribution of usage. 

When analyzing gender, there is no significant difference between the usage patterns or perfor¬ 
mance between males and females (Figure 4a). Essentially both groups are performing at comparable 
levels and are positively impacted by their use of screencasts. 

'. Tables highlighting the details of the statistical analysis are included for academic background (major) only. 
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Fall 2008 and Winter 2009 


Mean No. of 

Mean Final 

Course Grade 
(%) 

Number of 


Website Hits 

Students 

Sex 




Males 

23.13 

82.81 

295 

Females 

23.37 

83.50 

102 

Total 

23.19 

82.98 

397 

Citizenship Status 




U.S. Citizen (U.S.) 

22.33 

82.84 

340 

Non-U.S. Citizen (Non) 

28.33 

83.84 

57 

Total 

23.19 

82.98 

397 

Race/Ethnicity 




Caucasian (Cauc.) 

21.80 

83.13 

237 

Asian 

24.63 

83.31 

67 

Historically Underrepresented Minorities (URM) 4 

20.64 

78.76 

39 

Total 

22.22 

82.66 

343 

Academic Level 




Sophomore (Soph) 

18.04 

79.52 

85 

Junior (Jr.) 

24.87 

84.71 

176 

Senior (Sr.) 

24.33 

83.01 

135 

Total 

23.22 

83.02 

396 

Major 




Aerospace (AERO) 

20.49 

83.38 

96 

Chemical (ChE) 

19.03 

85.94 

94 

Nuclear Eng. & Radiological Sciences (NERS) 

16.00 

84.64 

28 

Industrial and Operations (IOE) 

32.14 

82.24 

113 

Material Science (MSE) 

28.36 

80.15 

14 

Other Engineering (Mechanical, Civil, etc) 

19.29 

78.47 

48 

Other (Non-Engineering) 

12.25 

77.38 

4 

Total 

23.19 

82.98 

397 


Table 3: Means for screencast usage (number of website Hits) and course performance 
(final course grade) across background characteristics in Fall 2008 and Winter 2009. 


Examining citizenship using one-way analysis of variance shows that Non-U.S. Citizens (perma¬ 
nent residents and non-residents) used screencasts at significantly higher rates than U.S. Citizens 
[F (1, 395) = 3.96, p = .047], with an average of 28.33 website hits per student compared to 22.33 
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Figure 4: Level of screencast use by (a) Gender, (b) Citizenship, (c) Race/Ethnicity and 
(d) Academic level for both terms. Level of screencast use are defined as follows: non-users 
(N) with O website hits, very low users (VL) with 1-iO website hits, low users (L) with 17-20 
website hits, mid-level users (M) with 21-30 website hits, high users (H) with 31-40 website 
hits, and very high users (VH) with more than 41 website hits per term. Gender is defined 
as follows: females (F) and males (M). Citizenship status is defined as follows: Non-U.S. 
citizens comprising of permanent residents and non-residents (Non), and U.S. Citizens 
(U.S.). Race/ethnicity are defined as follows: historically under-represented students in 
engineering comprising of African Americans, Hispanics, and Native Americans (URM); 
Asians (Asian); and Caucasians (Cauc.). Academic levels are defined as follows: sophomores 
(Soph.), juniors (Jr.), and seniors (Sr.). 
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website hits per student respectively. (See Table 3). Despite this significant difference in mean 
screencast use, similar analysis revealed no significant difference for mean academic performance 
when comparing U.S. Citizens and non-citizens [F (1, 395) = .52, p = .471], While both groups are 
performing on par with one another, the higher screencast use by Non-U.S. Citizens suggests that, for 
this group, screencast use may be the vehicle by which performance is raised to a level comparable 
to their U.S. Citizen counterparts. Certainly, both groups are benefitting from screencast use as each 
maintains a significant positive correlation between number of website hits and performance (r = 
.228, p = .000 and r = .342, p = .009 for U.S. Citizens and Non-U.S. Citizens respectively). While the 
slightly higher magnitude of the correlation for non-citizens implies that perhaps this group may 
be benefitting from screencast use a bit more than U.S. Citizens and thus should be encouraged 
to continue to use at higher rates, this difference is only slight and therefore should be interpreted 
with caution. 

We examined differences in mean screencast usage and mean course performance across three 
racial/ethnic categories—Caucasian, Asian, and those racial/ethnic groups that have been histori¬ 
cally under-represented (URM) in the College of Engineering. These include African American, 
Hispanic, and Native American students. Those students who marked “None” or did not indicate a 
racial/ethnic identity are excluded from these analyses and comprise 13% of the student population. 
While it appears that Asians use screencasts at slightly higher levels than Caucasians and URMs 
(see Table 3), one-way analysis of variance finds no significant difference in mean usage between 
these groups [F (2, 340) = .64, p = .526], In contrast, similar analysis reveals significantly differ¬ 
ent means for performance [F (2, 340) = 3.52, p = .031], due primarily to the difference between 
the higher average performance of Caucasians (83.13) compared with that of URMs (78.76) (p = 
.027). Essentially, comparable usage of screencasts is paired with lower course performance for 
URMs in relation to both Asians and Caucasians, begging the question of whether performance 
might be similar between the groups if URMs were to increase their screencast usage and pre¬ 
sumably raise their performance. We find however, that Caucasians are the only group to retain 
significance for the positive correlation between screencast use and course performance (r = .251, 
p = .000). The lower course performance of URMs appears to be unrelated to screencast use— 
the significant relationship between being a URM and lower course performance is retained even 
after controlling for number of website hits (r = -.139, p = .006). Rather, the lower course perfor¬ 
mance of URMs is mediated by GPA—when this measure is controlled for, the significance of the 
relationship between being a URM and receiving a lower final course grade disappears completely 
(r = -.004, p = .929). 

We also tested means for usage and performance across academic level (sophomore, junior, and 
senior) using one-way analysis of variance and found significant differences in both website hits 
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[F (2, 393) = 3.31, p = .038] * 2 and final course grade [F (2, 393) = 8.60, p = .000], 3 Sophomores 
used screencasts at significantly lower levels than juniors (p = .015) and seniors (p = .044)—means 
of 18.04, 24.87, and 24.33 respectively. As well, sophomores performed at significantly lower levels 
than both juniors (p = .000) and seniors (p = .022)—means of 79.51, 84.71, and 83.01 respectively. 
Essentially, being a sophomore has a negative effect on both usage and performance, suggesting 
the potential for increased usage to result in higher performance. Flowever, when the overall posi¬ 
tive, significant correlation between usage and performance is broken down by academic level, 
significance is retained by juniors (r = .194, p = .010) and seniors (r = .309, p = .000), but not 
sophomores (r = .184, p = .091). Thus, it appears that even sophomores who are using screencasts 
at higher levels are not experiencing the same performance benefits as are their junior and senior 
counterparts. Indeed, the negative relationship between sophomore status and performance retains 
its significance even after controlling for number of website hits (r = -.162, p = .001). In contrast, 
the significance of this relationship disappears when GPA is controlled for (r = -.055, p = .273), 
meaning that the relationship between being a sophomore and receiving a lower final course grade 
seems to be mediated by GPA and not screencast use. 


Usage and Performance by Academic Major 

Analyses of screencast use and performance according to student major illustrate differential 
use of screencasts by particular students. We focus our analysis on the three majors that comprise 
the largest portion of the student population; namely, Aerospace Engineers (AERO), Chemical En¬ 
gineers (ChE), and Industrial and Operations Engineers (IOE). 4 Our results show the potential for 
screencasts to impact course performance differently depending on students’ academic preparation/ 
backgrounds. This finding suggests that students who enter the course with academic backgrounds 
(majors) that have provided them with experiences and knowledge of the concepts to be covered in 
MSE 220 may perform comparatively better in the course, since they are already quite familiar with 
course material (Recall Figure 2). Thus, these students may be predisposed to use the screencasts 

Significance was also found using robust tests of equality of means. Welch robust F test: F (2, 236.405) = 4.598, p = .011 and 

Brown-Forsythe robust F test: F (2, 375.439) = 3.630, p = .027. These tests were run to determine significance given violation of 
the assumption of equality of variance at the p < .05 level: a Levene statistic of (2, 393) 3.64, p = .027. 

3 One freshmen was excluded from the analyses in Winter 2009 due to insufficient sample size. 

4 ln order to insure adequate sample size for comparative purposes and to satisfy normality and equality of variance assump¬ 
tions, the major groups with sample sizes below twenty were excluded from statistical analyses. Additionally, MSE students 
were not included due to the fact that the University of Michigan College of Engineering offers/recommends an alternate course 
specifically for MSE students that fulfills the introduction to MSE requirement. As such, the small number of MSE students who 
opt to take MSE 220 are unlikely to be representative of MSE majors in general (most of whom are enrolled in MSE 250). Finally, 
Other Engineers were excluded for theoretical reasons. Meaningful interpretation of results is hindered by variability in academic 
background among constituent majors (Mechanical Engineers, Electrical Engineers, etc.). 
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less frequently. In turn, students whose academic backgrounds have prepared them the least for the 
course content may use screencasts the most, perhaps due to their lack of familiarity and confidence 
related to the concepts, and that screencast usage results in better performance overall. 

This finding needs to be placed in context of student performance by academic major prior to 
the introduction of screencasting. Analysis of student performance in Fall 2006 shows that students 
across academic majors performed differently, with lOEs performing least favorably among all the 
major groups. One-way analysis of variance indicates significant differences in means for final course 
grade across each of the three largest major groups [F (2,113) = 4.04, p = .020]. (See Table 4 & 5 in 
Appendix). In this case, post-hoc tests show that this significance is attributable to the differences 
between lOEs and ChEs (p = .015), while no significance is found for AERO students in relation to 
the other major groups at the p < .05 level. (See Table 6 in Appendix). 

In terms of course performance for Fall 2008 and Winter 2009, Figure 5a illustrates the number 
of students in each major who fall into each of five grade categories (ranging from F through A). 
ChE students tend to concentrate at the high end of the spectrum receiving a mean grade of 86% 
(Table 3). In fact, ChE students perform the best of all three major groups (see Table 4 in Appen¬ 
dix), but this mean course grade is only statistically significant (p=.005) in relation to IOE students 
(82%) based on the Tukey post hoc test. Comparisons between IOE and AERO and between AERO 
and ChE were not significant at the .05 level. (See Tables 5 & 6 in Appendix for full ANOVA results.) 
It is important to note that AERO, ChE, and IOE students have comparable mean GPAs when they 
enter MSE 220, despite this difference in course performance. 

Screencast use is distributed variably, yet certain patterns emerge when focusing on the three larg¬ 
est major groups. lOEs comprise the bulk of very high users (Figure 5b) and are using screencasts at 
significantly higher rates than ChE or AERO students with mean number of website hits of 32.14,19.03, 
and 20.49 respectively (Table 3). In contrast, ChE and AERO students are more likely to be very low 
and low users (Figure 5b). One-way analysis of variance shows a significant difference in the means 
for the number of website hits between the three largest majors [F (2, 300) = 12.35, p = .000] 5 — 
attributable to the difference in usage between IOE and AERO students (p = .001) and IOE and ChE 
students (p =.000) based on the Games-Howell test. (See Tables 7 & 8 in Appendix.) Comparisons 
between ChE and AERO students were not significant at the .05 level. Essentially, IOE students were 
significantly more likely to have higher screencast use compared to students in other majors. 

Interestingly, IOE students who are most likely to use screencasts at high levels, tend to receive 
grades in the middle range when all majors are compared (Table 3). This is particularly noteworthy 


Significance was also found using robust tests of equality of means. Welch robust F test: F (2,198.480) = 10.529, p = .000 and 
Brown-Forsythe robust F test: F (2, 277.955) = 13.006, p = .000. These tests were run to determine significance given violation of 
the assumption of equality of variance at the p < .05 level: a Levene statistic of (2, 300) = 11.219, p = .000. 
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Figure 5: (a) Final letter grade and (b) level of screencast use by Academic Major. 
Screencast usage levels are defined as follows: non-users (N) with O website hits, very low 
users (VL) with 1-10 website hits, low users (L) with 11-20 website hits, mid-level users 
(M) with 21-30 website hits, high users (H) with 31-40 website hits, and very high users 
(VH) with more than 41 website hits per term. Majors are defined as follows: Aerospace 
Engineering (AERO), Chemical Engineering (ChE), Industrial and Operations Engineering 
(lOE), Materials Science Engineering (MSE), Nuclear Engineering and Radiological Sciences 
(NERS). 


given their low academic performance prior to the introduction of screencasts in Fall 2006 and 
the relative dissimilarity in course requirements for lOE students in comparison to other majors 
(Figure 2). On the other hand, ChE students performed the best of all three majors regardless of the 
availability of screencasts. Therefore, the supplementary screencasts are most helpful to students 
whose academic background is dissimilar to Materials Science. 

These results show that lOEs perform on par with fellow AERO and ChE students (despite their 
lack of prior exposure to course material) at least in part because of their higher screencast use. In 
examining the impact of the level of screencast use (ranging from very low to very high) on final 
course grade, we find that lOEs are the only major group for which course performance differs 
significantly between different usage levels, meaning that for these students high usage is signifi¬ 
cantly related to high course performance. A one-way analysis of variance for each of the three 
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largest major groups shows significance for IOE students [F (4,101) = 6.619, p = .000] 6 (see Table 
9 in Appendix), while Games-Howell post-hoc results attribute this to the difference in course per¬ 
formance means between the very low usage group and the very high usage group (p = .035). The 
means for all other usage groups were not significant at the .05 level. (See Table 10 in Appendix). 
This indicates that lOEs who are using screencasts the least are significantly more likely to receive 
lower grades, while those using screencasts the most are significantly more likely to receive higher 
grades. In a similar test for AEROs, despite the initial significance of the course performance means 
across the various screencast usage levels [F (4, 83) = 2.680, p = .037], Tukey post-hoc tests in¬ 
dicate no significance for any of the usage levels in relation to each other. As well, Games-Flowell 
comparisons show only slight significance between the means for very low users and very high 
users (p = .52), with no significant differences in means for any of the other levels of usage at the 
p < .05 level. (See Table 11 in Appendix). This suggests that AEROs who use screencasts at very low 
levels are somewhat more likely to receive lower grades and vice versa, however this significance 
was only slight and should therefore be interpreted with considerable caution. Finally, no signifi¬ 
cance was found between performance means across levels of screencast use for ChE students. For 
these students, screencast use appears to be unrelated to course performance. Indeed, the positive 
relationship between ChE status and performance retains its significance even after controlling for 
both the number of website hits and cumulative GPA (r = .215, p = .000). 

Summary 

In this work, various types of screencasts were developed to address student needs in an intro¬ 
ductory lecture course in Materials Science. Homework solution screencasts and “muddiest point” 
screencasts were used the most by students, and screencast usage was positively and significantly 
correlated to overall performance in the course. In other words, those students who used screencasts 
more earned a higher grade in the course. When we examine differences in means for screencast 
usage and course performance across the dimensions of gender, citizenship, race/ethnicity, aca¬ 
demic level, and major; the results are mixed. For both gender and citizenship, students appear to 
benefit from screencast use more or less equally, regardless of whether they are male or female, 
U.S. Citizens or not. In terms of race/ethnicity and academic level, certain groups do demonstrate 
different relationships between screencast use and performance. However, for the two groups who 
stand to gain the most in terms of final course grade (sophomores and URMs), we find that GPA, 
rather than screencast use, provides the best explanation for course performance. 


Significance was also found using robust tests of equality of means. Welch robust F test: F (4, 39.720) = 3.632, p = .013 and 
Brown-Forsythe robust F test: F (4, 41.433) = 5.465, p = .001. These tests were run to determine significance given violation of 
the assumption of equality of variance at the p < .05 level: a Levene statistic of (4,101) = 5.606, p = .000. 
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In contrast, there was a significant difference in screencast usage and performance across aca¬ 
demic major, with ChE students and IOE students standing out among the major groups. Those 
students most prepared to encounter and subsequently master MSE 220 course content (the 
Chemical Engineers) use screencasts at low levels, while receiving the highest grades in the course. 
Indeed, for ChE students, no significant differences in means for final course grade are found across 
levels of screencast use and the positive relationship between being a ChE and receiving higher 
final course grades is not mediated by either usage or GPA. It appears that these students stand to 
benefit the least from screencast use. Quite the opposite of ChE students, IOE students begin the 
semester with the least familiarity with course content, use the screencasts the most, and receive 
average grades in the course. It is remarkable that the students who enter with the least preparation 
in Material Science are not receiving the lowest grades in the course, which was the case prior to 
the introduction of screencasts. Looking across the various semesters, it appears that IOE students 
have the most to gain by using the screencasts, since they start out at a disadvantage in terms of 
academic preparation for the course. Indeed, once screencasts became available and lOEs began 
to use them at high rates, this group improved their collective performance remarkably. 


IMPLICATIONS AND FUTURE WORK 


Our research shows the positive impact of screencasts as a supplementary resource to aid student 
learning, especially for those whose academic background may have provided them with less familiar¬ 
ity with course content than other students despite having similar overall grade point averages. We 
also find that other demographic dimensions, such as gender, citizenship, ethnicity, and academic 
level, are not significant indicators for the relationship between use and performance. However, 
interesting trends emerge that bear further study. For instance, while we find that Non-U.S. Citizen 
use screencasts at high rates and sophomores use at low rates, the reasons remain unclear. 

Some researchers have noted the impact of academic preparation on student performance [26]- 
[30]. In particular, Zhanf and RiCharde showed that academic preparation, study skills, and the college 
environment are correlated with academic achievement in freshman; however, academic motivation 
and self efficacy were not [30], Our research extends this finding by demonstrating that students 
who enter a course with comparable academic ability, but whose academic backgrounds prepared 
them less for the course content, showed gains in performance based on their use of screencasts. 
Seifert et al. [27] found that students who have the lowest academic preparation, parental educa¬ 
tion, involvement in high school prior to college, and academic motivation experienced the highest 
benefits as a result of good practices in undergraduate education as described by Chickering and 
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Gamson [31], Using screencasts as supplementary resources is an example of one of Chickering and 
Gamson’s principles in that these resources respects diverse talents and ways of learning by allowing 
students the flexibility to study concepts at their own pace and in more detail. 

In all, the development of screencasts for a large lecture survey course has the potential to “level 
the playing field,” and may be applied to courses where there is a wide range in student academic 
background knowledge such as introductory computer science courses, design courses, and even 
laboratory courses where students may enter with varying levels of familiarity with experimentation 
and writing lab reports. Further, this research used a variety of screencasts to aid student learning. 
Faculty who want to strategically develop their supplementary resources should consider creating 
“muddiest point” screencasts in courses comprised of students from a variety of academic back¬ 
grounds; while homework solution screencasts are applicable to any course where step by step 
explanations of procedures are necessary. Since students perceive all screencasts to be helpful, 
their development offers benefits above and beyond traditional text-based resources, as they can 
synthesize many resources, and provide a multimedia presentation of concepts that can reach a 
wider range of learners. 
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APPENDIX 


95% Confidence 
Interval for Mean 



Major 

N 

Mean 

SD 

SE 

Lower 

Upper 

Fall 2006 

AERO 

41 

81.41 

5.88 

.92 

79.56 

83.27 


ChE 

21 

85.27 d 

6.48 

1.41 

82.32 

88.22 


NERS 

11 

83.22 

5.81 

1.75 

79.32 

87.12 


IOE 

54 

80.17 

7.88 

1.07 

78.02 

82.32 


MSE 

16 

81.69 

6.09 

1.52 

78.44 

84.93 


Other Eng. (Mech., Civil, etc.) 

23 

83.38 

7.06 

1.47 

80.33 

86.44 


Other (Non-Eng.) 

1 

- 

- 

- 

- 

- 


Total 

167 

81.91 

6.94 

.54 

.80.85 

82.97 

Fall 2008 & 

AERO 

96 

83.38 

8.98 

.92 

81.56 

85.20 

Winter 2009 

ChE 

94 

85.94 

7.30 

.75 

84.44 

87.44 


NERS 

28 

84.64 

6.90 

1.30 

81.98 

87.31 


IOE 

113 

82.24 

9.51 

.89 

80.47 

84.01 


MSE 

14 

80.15 

18.57 

4.96 

69.43 

90.87 


Other Eng. (Mech., Civil, etc.) 

48 

78.47 

11.36 

1.64 

75.17 

81.77 


Other (Non-Eng.) 

4 

77.38 

11.11 

5.55 

59.70 

95.05 


Total 

397 

82.98 

9.69 

.49 

82.03 

83.94 


Table 4: Mean Performance (Final Course Grade) by Student Academic Background 
(Major). 


ANOVA 

Between 
Groups df 

Within 
Groups df 

F 

sig 

Fall 2006 

2 

113 

4.04 

p = .020 

Fall 2008 & 

Winter 2009 

2 

300 

4.76 

p = .009 


Table 5: Summary of ANOVA of Means for Course Performance (Final Course Grade) by 
Student Academic Background (Major). 
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95% Cl 



Mean Weight 

Std. 

Lower 

Upper 


Comparisons 

Difference (kg) 

Error 

Hound 

Bound 

Fall 2006 

AERO vs. ChE 

-3.85 

1.87 

-8.31 

.60 


IOE vs. AERO 

-1.24 

1.45 

-4.68 

2.19 


IOE vs. ChE 

-5.10* 

1.80 

-9.36 

-.83 

Fall 2008 & 

AERO vs. ChE 

-2.56 

1.26 

-5.54 

.42 

Winter 2009 

IOE vs. AERO 

-1.14 

1.21 

-3.99 

1.71 


IOE vs. ChE 

-3.70** 

1.22 

-6.57 

-.84 

* p < .05, ** p 

< .01 






Table 6: Tukey Post-hoc Comparison of Mean Course Performance (Final Course Grade) by 
Student Academic Background (Major). 



Between 

Within 



ANOVA 

Groups df 

Groups df 

F 

sig 

Fall 2008 & 
Winter 2009 

2 

300 

12.35 

p = .000 


Table 7: Summary of ANOVA of Means for Screencast Usage (Number of Website Hits) by 
Student Academic Background (Major). 





95% Cl 

Comparisons 

Mean Weight 
Difference (kg) 

Std. 

Error 

Lower 

Bound 

Upper 

Bound 

AERO vs. ChE 

1.46 

2.54 

-4.54 

7.45 

IOE vs. AERO 

11.65** 

3.11 

4.30 

19.00 

IOE vs. ChE 

13.11** 

2.94 

6.17 

20.05 


** ^ << m 


Table 8: Games-Howeii Post-hoc Comparison of Mean Screencast Usage (Number of 
Website Hits) by Student Academic Background (Major) in Fall 2008 and Winter 2009. 
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ANOVA 


Between 
Groups df 

Within 
Groups df 

F 

sig 

Fall 2008 & 
Winter 2009 

IOE 

4 

101 

6.62 

p = .000 

AERO 

4 

83 

2.68 

p = .038 


CHE 

4 

85 

0.44 

NS 


Table 9: Summary of ANOVA of Means for Course Performance (Final Course Grade) by 
Screencast Usage (Level of Screencast Use) for Aerospace, Chemical and Industrial and 
Operations Engineers in Fall 2008 and Winter 2009. 





95% Cl 

Comparisons 

Mean Weight 
Difference (kg) 

Std. 

Error 

Lower Bound 

Upper Bound 

Very Low (1-10) vs. Low (11-20) 

-10.50 

4.18 

-22.95 

1.95 

Very Low (1-10) vs. Mid (21-30) 

-10.39 

4.12 

-22.68 

1.90 

Very Low (1-10) vs. High (31-40) 

-6.72 

4.55 

-20.05 

6.62 

Very Low (1-10) vs. Very High (41+) 

-12.78* 

3.98 

-24.83 

-.734 

Low (11-20) vs. Mid (21-30) 

.11 

2.10 

-5.91 

6.13 

Low (11-20) vs. High (31^10) 

3.78 

2.88 

-4.69 

12.26 

Low (11-20) vs. Very High (41 +) 

-2.28 

1.84 

-7.60 

3.03 

Mid (21-30) vs. High (31-40) 

3.68 

2.76 

-4.54 

11.89 

Mid (21-30) vs. Very High (41+) 

-2.39 

1.65 

-7.15 

2.37 

High (31-40) vs. Very High (41+) 

-6.07 

2.57 

-13.88 

1.75 


*p < .05 


Table 10: Games-Howell Post-hoc Comparison of Mean Performance (Final Course Grade) 
by Level of Screencast Use for Industrial and Operations Engineers in Fall 2008 and Winter 
2009. 


Comparisons 

Mean Weight 
Difference (kg) 

Std. 

Error 

95% Cl 

Lower 

Bound 

Upper 

Bound 

Very Low (1-10) vs. Low (11-20) 

-1.48 

2.31 

-8.07 

5.11 

Very Low (1-10) vs. Mid (21-30) 

-.17 

2.53 

-7.42 

7.08 

Very Low (1-10) vs. High (31-40) 

-5.91 

2.60 

-13.35 

1.53 

Very Low (1-10) vs. Very High (41+) 

-6.16t 

2.18 

-12.36 

.03 

Low (11-20) vs. Mid (21-30) 

1.31 

2.23 

-5.28 

7.90 

Low (11-20) vs. High (31^40) 

-4.43 

2.32 

-11.21 

2.35 

Low (11-20) vs. Very High (41 +) 

-4.69 

1.83 

-10.05 

.68 

Mid (21-30) vs. High (31-40) 

-5.74 

2.53 

-13.14 

1.66 

Mid (21-30) vs. Very High (41 +) 

-5.99 

2.09 

-12.21 

.22 

High (31-40) vs. Very High (41 + ) 

-.25 

2.18 

-6.67 

6.16 


t p < .10 


Table 11: Games-Howell Post-hoc Comparison of Mean Performance (Final Course Grade) 
by Level of Screencast Use for Aerospace Engineers in Fall 2008 and Winter 2009. 
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